Software product quality is increasingly important for competitive advantage in commercial software development. A number of industrial benchmarks exist that consumers utilize in making buying decisions. Meeting or exceeding such benchmarks requires software design that is effective, reliable, and fast. Monitoring the performance of computer programs under development is an important aspect of improving software product quality.
Monitoring performance of software under development should be frequent and effective. In the typical software development environment, change to the source code is ongoing and developers working on isolated subsystems are often unaware of whether their changes adversely affect performance of the larger system. Compounding this is the fact that many different hardware processors, such as a Central Processing Unit (CPU) and Graphics Processing Unit (GPU), are supported and each may have their own unique performance properties. In addition, modern system and driver designs attempt to maximize utilization of the processor(s) through multitasking and caching techniques, which in turn results in random variability of performance. If a serious performance regression is introduced and not discovered until a much later time, then debugging the initial cause of the regression may be a difficult and expensive task.
Available techniques for monitoring software performance are slow and difficult. Due to a huge number of possible hardware configurations in modern computing, the practical problem of identifying and analyzing regressions in software performance on a wide range of hardware configurations is a daunting task. Test results can proliferate. The amount of time required to manually analyze the results to identify regressions and their probable causes is impractical, often resulting in less frequent and therefore less effective product testing. Automated techniques are necessary to process the sheer number of tests in a comprehensive test pass, so that meaningful regressions can be identified without painstaking manual analysis.